Systems and Methods for Reducing Data Storage Overhead by Utilizing Rolling Indices

ABSTRACT

Systems and methods for reducing data storage overhead are disclosed herein. In some embodiments, a system includes a rollup service that converts a raw data set into a rolled up index that takes up less storage than the raw data but is created in such a way that the rolled up index can be queried so as to generate responses that will substantially correspond to responses that would be generated using the raw data.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.16/175,721, filed Oct. 30, 2018, entitled “Systems and Methods forReducing Data Storage Overhead,” the disclosure of which is incorporatedby reference for all purposes.

FIELD

The present disclosure is directed to systems and methods that reducedata storage overhead with respect to large volumes of raw historicaldata by generating rollup indices that summarize the raw historical dataaccording to aggregation trees. These systems and methods also allow forquery execution of the rollup indices and/or the raw data in anefficient manner for creation of graphical or visual data depictions.

SUMMARY

According to various embodiments, the present technology is directed toa method comprising: obtaining raw data from one or more systemsproviding one or more services, the raw data comprising data elementshaving corresponding fields and metrics, the data elements beingobtained according to a first time interval, the raw data having astorage size; generating a rollup index of the raw data based on a firstset of rollup parameters that comprise selections of the plurality offields and metrics by: searching for matching ones of the data elementsof the raw data that correspond to the first set of rollup parameters;grouping the data elements together based on the matching; flatteningthe data elements by rolling up the data elements into a document basedon a second time interval that is larger than the first time interval;and storing the rollup index in a storage space, the rollup index beingreduced in size relative to the raw data so as to reduce an amount ofthe storage space required to store rollup index relative to storing theraw data.

According to various embodiments, the present technology is directed toa system comprising: a rollup service configured to: obtain raw datafrom one or more systems providing one or more services, the raw datacomprising data elements having corresponding fields and metrics, thedata elements being obtained according to a first time interval, the rawdata having a storage size; generate a rollup index of the raw databased on a first set of rollup parameters that comprise selections ofthe fields and metrics by: searching for matching ones of the dataelements of the raw data that correspond to the first set of rollupparameters; grouping the data elements together based on the matching;flattening the data elements; and rolling up the data elements forplacement into the rollup index based on a second time interval that islarger than the first time interval; and store the rollup index in astorage space, the rollup index being reduced in size relative to theraw data so as to reduce an amount of the storage space required tostore rollup index relative to storing the raw data; and a searchservice endpoint configured to: receive a query having query parameters;parse the query parameters; search the query parameters against therollup index; and generate a response using the rollup index.

According to various embodiments, the present technology is directed toa method comprising: obtaining raw historical data for a computingsystem, the raw historical data being according to a first timeinterval; converting the raw historical data into a rollup index, therollup index comprising aggregations of data elements of the rawhistorical data which are grouped according to at least one field and atleast one metric regarding the at least one field; and generating aquery response for a query using the rollup index, the query responsesubstantially corresponding to a query response when executed againstthe raw historical data due to the conversion of the raw historical datainto the rollup index.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, where like reference numerals refer toidentical or functionally similar elements throughout the separateviews, together with the detailed description below, are incorporated inand form part of the specification, and serve to further illustrateembodiments of concepts that include the claimed disclosure, and explainvarious principles and advantages of those embodiments.

The methods and systems disclosed herein have been represented whereappropriate by conventional symbols in the drawings, showing only thosespecific details that are pertinent to understanding the embodiments ofthe present disclosure so as not to obscure the disclosure with detailsthat will be readily apparent to those of ordinary skill in the arthaving the benefit of the description herein.

FIG. 1 is a schematic diagram of an example system constructed inaccordance with the present disclosure.

FIG. 2 is a diagrammatic representation of the use of aggregation treesto flatten data and coalesce rollup jobs into a rollup index.

FIG. 3 is a flowchart of an example method for identifying rollup jobsof a rollup index using a recursive process.

FIG. 4 is a flowchart of an example method of the present disclosure.

FIG. 5 is a schematic diagram of an example computer device that can beutilized to implement aspects of the present technology.

DETAILED DESCRIPTION

The present disclosure relates generally to systems and methods thatstore raw historical data in a condensed but content rich format. Thesesystems and methods can reduce large volumes of raw historical data intocompact but useful rollup jobs and indices. A rollup index of thepresent disclosure can be queried against in a manner that is similar oridentical to query methods for the raw historical data, for purposes ofdata analysis and visualization.

For context, in time-series storage and visualization systems, rollupsare defined by reports and dashboards (e.g., graphical user interfaces)they are used to generate. This sort of pre-definition means that usersare stuck with the same report even if the users need to change thereport format and/or content. Such configurations are unduly limiting.Advantageously, the systems and methods disclosed herein do notsacrifice search speed even in view of a large data volume. To be surethere is substantial value in keeping the original, full, raw dataallowing a user to analyze and parse it however they desire, withoutcompromise, but this can lead to deleterious effects and cost.

High-level dashboards may need decades worth of data, but likely notdecades worth of fine-grained data that takes up considerable diskspace. The rollup functionalities disclosed herein allow users to pickall the fields they want rolled up and a new rollup index is createdwith aggregated portions of the raw data. This new rollup index thenresides side-by-side (in some embodiments) with the raw data that it isbe being rolled up from. Since rollups are functionally and conceptuallysimilar to any other raw data index, only a lot smaller, users arealready knowledgeable regarding how to query and aggregate the datainside. The systems and methods disclosed herein also implement a rollupAPI (application programming interface) that provides an ability tosearch both live (e.g., raw) and rollup data at the same time, returningdata from both indices in a single response.

When raw data is retired to permanent storage any corresponding rollupindex created therefrom can live on with other production indexes.Dashboard creation from rollup indices are not affected by these changesand retirement of old raw data. These processes are also very flexible.For example, if a user realizes a mistake in data collection has beenmade in a rollup, the user can direct a system of the present disclosureto pull an old raw data index back into production and run a new andimproved rollup job.

Time-based data (documents that are predominantly identified by theirtimestamp) often have associated retention policies to manage datagrowth. For example, a system may be generating 500,000 documents everysecond. That will generate 43 million documents per day, and nearly 16billion documents a year.

When these processes continue in perpetuity, storage requirements willcontinue to grow without bound. Retention policies are therefore oftendictated by the simple calculation of storage costs over time, and whatan organization is willing to pay to retain historical data. Often thesepolicies require deleting data after a few months or years.

To be sure, storage cost is a fixed quantity. That is, it requires Xmoney to store Y data, but the utility of a piece of data often changeswith time. For example, sensor data gathered at millisecond granularityis extremely useful when generated, reasonably useful if a few weeksold, and only marginally useful if older than a few months. So while thecost of storing a millisecond of sensor data from ten years ago isfixed, the value of that individual sensor reading often diminishes withtime. It is not useless as it could easily contribute to a usefulanalysis, but this reduced value often leads to deletion rather thanpaying the fixed storage cost.

Thus, the systems and methods disclosed herein rollup store historicaldata at reduced granularity to reduce storage overhead of data, whileallowing users to analyze and assess the data that is most important totheir needs. That is, data collected in time intervals of seconds can beaggregated into blocks of time such as hours or days, for example.

A rollup functionality as disclosed herein summarizes old,high-granularity raw data into a reduced granularity format forlong-term storage. By “rolling” the data up into a single summarydocument, historical data can be compressed greatly compared to the rawdata. For example, consider a system that is generating 43 milliondocuments every day. The second-by-second data is useful for real-timeanalysis, but historical analysis looking over ten years of data arelikely to be working at a larger interval such as hourly or dailytrends. An example system of the present disclosure will compress the 43million documents into hourly summaries saving vast amounts of storagespace. The rollup feature automates this process of summarizinghistorical data. These and other aspects and advantages of the presentdisclosure are provided herein with reference to the collectivedrawings.

FIG. 1 is an example schematic diagram of an architecture for practicingaspects or embodiments of the present disclosure. In one embodiment, oneor more systems 100A-N are generating a large volume of documents102A-Z. A rollup server or cloud service 104 (generally rollup service)generates rollup indices, such as rollup index 106, from at least aportion of the raw data included in the large volume of documents102A-Z. This rollup service 104 can be implemented as an API in someembodiments.

Also, a raw index 108 of the large volume of documents 102A-Z can becreated and stored as well. Both the rollup index 106 and the raw index108 can reside in a database 110. The rollup index 106 and/or raw index108 can be queried using a search service endpoint 112 that can beco-located with the rollup server or cloud service 104. A user 114 canexecute queries against the database 110 using the search serviceendpoint 112 and receive results in the form of reports or graphicalinterfaces such as histograms or data plots. In general, the componentsof the architecture can communicate through an example network 116 thatcan include any public and/or private network.

The documents 102A-Z can comprise any documents such as log files. Forexample, the documents can include network activities of devices with aspecific IP addresses. Other example documents can include cloud computeresource availability and/or utilization, web service usage, query orsearching services, real-time sensor data, as well as any othercomputer-implemented process or system that produces quantifiable orqualitative data. The raw data of these documents 102A-Z can beexpressed in terms of fields and metrics. That is, the raw data iscomprised of individual data elements. Each of the data elementscomprises at least one field and at least one metric related to the atleast one field. For example, field related to IP address can be used toidentify every IP address that logs into a particular system or service.A metric relates to attributes of that field. For example, with respectto IP addresses, metrics could include a minimum or maximum amount ofbandwidth used by each IP address tracked. Again, these are merelyexamples and specific details regarding fields and metrics are governedby the systems and/or services used to produce the raw data. As notedabove, these raw data can be voluminous and consume a drastic amount ofdata storage.

These raw data can be collected at a specific interval, such as secondsor even milliseconds. This is referred to generally to as a first timeinterval. This first time interval represents the datacollection/generation speed of the one or more systems 100A-N.

The rollup index 106 is generated from the raw data of these documents102A-Z using the rollup service 104. In some embodiments, the means bywhich the rollup service 104 generates the rollup index 106 is in partdictated by the end user. That is, the end user can determine whichfields and/or metrics are of interest and select these during thecreation of a rollup job. In general, many rollup jobs can be initiatedif needed. In some instances, a rollup job runs continuously in thebackground as the documents 102A-Z are being generated. In a continualprocess, the rollup service 104 will roll up the index or indices thatare specified by the end user, placing the rolled documents in asecondary index (also of the user's choosing) such as the database 110.

In one non-limiting example, a series of data are provided in thedocuments 102A-Z which are indicative of sensor data (sensor-2017-01-01,sensor-2017-01-02, and so forth). Documents 102A-Z related to thesesensors can be collected using the following code:

{ “timestamp”: 1516729294000, “temperature”: 200, “voltage”: 5.2, “node”: “a” }

In general, the rollup service 104 will roll up these documents foundinto hourly summaries, which will allow the rollup service 104 togenerate reports and dashboards such as report 118 with any timeinterval that is one hour or greater. To be sure, in this example, thehourly summary reflected by the roll up represents a second timeinterval that is longer than the first time interval that is measured inseconds or milliseconds. Thus, data representative of data elementsgathered at second time intervals is summed up in the roll up index asan hourly summary. Again, these time intervals are examples and are notintended to be limiting.

An example roll up job is illustrated below:

{ “index_pattern”: “sensor-*”, “rollup_index”: “sensor_rollup”,  “cron”:“*/30 * * * * ?”  “page_size” :1000,  “groups” : { “date_histogram”: {“field”: “timestamp”, “interval”: “60m” }, “terms”: {  “fields”:[“node”]} },  “metrics”: [ {“field”: “temperature”, “metrics”: [“min”,“max”, “sum”]}, { “field”: “voltage”, “metrics”: [“avg”]} }}

The rollup service 104 assigns the rollup job an identifier of “sensor”and executes the rollup job to rollup the index pattern “sensor-*”. Thisjob will find and rollup any index data that matches that pattern.Rollup summaries are then stored in a rollup index titled“sensor_rollup”.

The cron parameter controls when and how often the job activates. When acron schedule triggers, it will begin rolling up from where it left offafter the last activation. So if the rollup job is configured such thatthe cron to run every 30 seconds, the job will process the last 30seconds worth of data that was indexed into the sensor-* indices.

If instead the cron was configured to run once a day at midnight, thejob would process the last 24 hours' worth of data. The choice islargely preference, based on how “real-time” the user wishes for therollups to be, and if the user wishes to process continuously or move itto off-peak hours.

Next, the rollup service 104 is utilized to define a set of groups andmetrics. In this example the metrics are selected to save themin/max/sum of a temperature field, and the average of the voltagefield. In this example, the groups are created by defining thedimensions that will be used to pivot on at a later date when queryingthe data. The grouping in this rollup job allows for the later creationof a function such as date_histograms aggregations on the timestampfield, rolled up at hourly intervals. It also allows the rollup service104 to run term(s) aggregations on a node field.

In this example, the cron is configured to run every 30 seconds, but thedate_histogram is configured to rollup at 60 minute intervals. In thisexample the date_histogram function controls the granularity of thesaved data. Data will be rolled up into hourly intervals, and the userwill be unable to query with finer granularity. The cron controls whenthe process looks for new data to rollup. Every 30 seconds it will seeif there is a new hour's worth of data and roll it up. If not, therollup job goes back to sleep.

Often, it may not make sense to define such a small cron (30 s) on alarge interval (1 h), because the majority of the activations willsimply go back to sleep. But there is nothing incorrect about thisselection. The rollup job is configured to intelligently functionregardless of the time interval selected.

After the rollup job is created, it will be sitting in an inactivestate. Thus, rollup jobs need to be started before they begin processingdata (this allows the user to stop them later as a way to temporarilypause, without deleting the configuration).

As noted above, many different rollup jobs can be performed for a set ofraw data. These rollup jobs can, in some instances, be partially orcompletely orthogonal in content to one another. For example, the aboveuse case of sensor data could be combined with sales and marketing dataregarding the sensors. Queries could be used to search throughperformance and sales data in single query, just as an example. Thesearch service endpoint 112 (discussed in greater detail herein) canstitch together query responses using rollup jobs that are orthogonal incontent to one another.

The query would include parameters directed to both sensor performanceand sales data, possibly to determine if sensor performance is affectingsales for that particular sensor. The search service endpoint 112 canparse these separate query parts are search different rollup jobs in asingle rollup index and concatenate the responses together in a singleresponse.

In some embodiments the rollup service 104 exposes a unique searchendpoint, such as the search service endpoint 112, which is used tosearch over rolled-up indexes and their corresponding rollup jobs.Importantly, this endpoint accepts normal Elasticsearch™ Query DSL.Thus, any application accessing the service does not need to learn a newDSL to inspect historical data; it can simply reuse existing queries anddashboards.

In certain embodiments, a rollup merges can merge live/raw and rolleddata together. In more detail, a useful feature of the rollup service isthe ability to query both “live”, real-time data in addition tohistorical rolled up data in a single query. For example, a system maykeep a month of raw data. After a month, it is rolled up into historicalsummaries using the rollup features described herein and the raw data isdeleted. If the user were to query the raw data, they would only see themost recent month. And if the user were to query the rolled up data, youwould only see data older than a month. The search service endpoint 112,however, supports querying both the raw index 108 and the rollup index106 at the same time. It will take the results from both data sourcesand merge them together. If there is overlap between the “live” and“rolled” data, live data is preferred to increase accuracy.

Finally, rollup service 104 is capable of intelligently utilizing thebest interval available. For example existing systems require a user toconfigure rollups to happen at daily intervals, any correspondingqueries and charts will work with daily intervals. If the user prefers amonthly interval, the user will create another rollup that explicitlystores monthly averages, and so forth.

Advantageously, the rollup features disclosed herein store data in sucha way that queries can identify a smallest available interval and usethat for their processing. If rollups are stored at a daily interval,queries can be executed on daily or longer intervals (weekly, monthly,and so forth) without the need to explicitly configure a new rollup job.This helps alleviate one of the major disadvantages of a rollup system;reduced flexibility relative to raw data. Again, the query intervalselected can be equal to or larger than the interval used to create therollups that are queried. Thus, if the rollup is created with a minuteinterval the query executed against the rollup should be parameterizedfor minute, day, week, month, or larger intervals.

The following descriptions provide additional detail on searching rollupindexes (and additionally raw indexes). After one or more rollup jobshave run and processed some data, the search service endpoint 112 willexecute a structured query. The rollup processes are configured so thatusers can use the same Query DSL syntax that they are accustomed tousing with the raw data with the exception that it is run on the rolledup data instead.

An example query is provided for context:

GET /sensor_rollup/_rollup_search { “size”: 0, “aggregations”: {“max_temperature”: { “max”: { “field”: “temperature” } } } }

This query resembles a simple aggregation that calculates the maximum ofthe temperature field. It will be understood that it is being sent tothe sensor_rollup index instead of the raw sensor-* indices. It will befurther understood that it is using the rollup_search endpoint. Resultsfrom the above query look like a normal aggregation response:

{ “took” : 102, “timed_out” : false, “terminated_early” : false,“_shards” : ... , “hits” : { “total” : 0, “max_score” : 0.0, “hits” : [] }, “aggregations” : { “max_temperature” : { “value” : 202.0 } } }

The only notable difference is that the rollup search results have zerohits, but is identical syntactically to a search that would be performedon raw data. To be sure, even though the data was rolled up with hourlyintervals and partitioned by node name, the query is calculating the maxtemperature across all documents. The groups that were configured in thejob are not mandatory elements of a query; they are just extradimensions that can be partition on. Second, the request and responsesyntax is nearly identical to standard DSL, making it easy to integrateinto dashboards and applications.

Finally, these grouping fields cab be used to construct a morecomplicated query:

GET /sensor_rollup/_rollup_search { “size”: 0, “aggregations”: {“timeline”: { “date_histogram”: { “field”: “timestamp”, “interval”: “7d”}, “aggs”: { “nodes”: { “terms”: { “field”: “node” }, “aggs”: {“max_temperature”: { “max”: { “field”: “temperature” } }, “avg_voltage”:{ “avg”: { “field”: “voltage” } } } } } } } } Which returns acorresponding response: {  “took” : 93,  “timed_out” : false, “terminated_early” : false,  “_shards” : ... ,  “hits” : {  “total” :0,  “max_score” : 0.0,  “hits” : [ ]  },  “aggregations” : {  “timeline”: {  “meta” : { },  “buckets” : [  {  “key_as_string” :“2018-01-18T00:00:00.000Z”,  “key” : 1516233600000,  “doc_count” : 6, “nodes” : {  “doc_count_error_upper_bound” : 0,  “sum_other_doc_count”: 0,  “buckets” : [  {  “key” : “a”,  “doc_count” : 2, “max_temperature” : {  “value” : 202.0  },  “avg_voltage” : {  “value”: 5.1499998569488525  }  },  {  “key” : “b”,  “doc_count” : 2, “max_temperature” : {  “value” : 201.0  },  “avg_voltage” : {  “value”: 5.700000047683716  }  },  {  “key” : “c”,  “doc_count” : 2, “max_temperature” : {  “value” : 202.0  },  “avg_voltage” : {  “value”: 4.099999904632568  }  }  ]  }  }  ]  }  } }

In addition to being more complicated (date histogram and a termsaggregation, plus an additional average metric), it is noteworthy thatthe date_histogram uses a 7 d (seven day) interval instead of 60 m(sixty minutes).

To preserve flexibility, a rollup job can be defined based on how futurequeries may need to use the data. Traditionally, systems force users tomake decisions about what metrics to rollup and on what interval. Forexample, an average of cpu_time on an hourly basis. This is limiting;if, at a future date, the user wishes to see the average of cpu_time onan hourly basis and partitioned by host_name, not such query can beexecuted.

In some instances, the user could decide to rollup the [hour, host]tuple on an hourly basis, but as the number of grouping keys grows, sodo the number of tuples the admin needs to configure. Furthermore, these[hours, host] tuples are only useful for hourly rollups. Daily, weekly,or monthly rollups would all require new configurations. The presentdisclosure also addresses these limitations. Rather than force the userto decide ahead of time which individual tuples should be rolled up, therollup jobs disclosed herein are configured based on which groups arepotentially useful to future queries. For example, this configuration:

“groups” : { “date_histogram”: { “field”: “timestamp”, “interval”: “1h”,“delay”: “7d” }, “terms”: { “fields”: [“hostname”, “datacenter”] },“histogram”: { “fields”: [“load”, “net_in”, “net_out”], “interval”: 5 }}

Allows date_histogram's to be used on the “timestamp” field, termsaggregations to be used on the “hostname” and “datacenter” fields, andhistograms to be used on any of “load”, “net_in”, “net_out” fields.Importantly, these aggs/fields can be used in any combination. Thisaggregation:

“aggs” : { “hourly”: { “date_histogram”: { “field”: “timestamp”,“interval”: “1h” }, “aggs”: { “host_names”: { “terms”: { “field”:“hostname” } } } } } is just as valid as this aggregation: “aggs” : {“hourly”: { “date_histogram”: { “field”: “timestamp”, “interval”: “1h”}, “aggs”: { “data_center”: { “terms”: { “field”: “datacenter” } },“aggs”: { “host_names”: { “terms”: { “field”: “hostname” } }, “aggs”: {“load_values”: { “histogram”: { “field”: “load”, “interval”: 5 } } } } }} }

It is noteworthy that the second aggregation is not only substantiallylarger, it also swapped the position of the terms aggregation on“hostname”, illustrating how the order of aggregations does not matterto rollups. Similarly, while the date_histogram is required for rollingup data, it is not required while querying (although often used). Forexample, this is a valid aggregation for a rollup search to execute:

“aggs” : { “host_names”: { “terms”: { “field”: “hostname” } } }

Ultimately, when configuring groups for a rollup job, the user cancontemplate how they might wish to partition data in a query at a futuredate and then include those in the configuration details of the rollupjob (e.g., fields and metrics selected). Because rollup searchfunctionality allows any order or combination of the grouped fields, theuser need only decide if a field is useful for aggregating, and how theymight wish to use it (terms, histogram, and so forth) in the future.

During searching, the search service endpoint 112 will break a querydown into parts through parsing. In some embodiments, each distinct partof a query will be used to search either the raw index 108 and/or therollup index 106.

In some embodiments, when querying raw data and rollup data, internallytwo search requests can be executed in parallel or series. The firstsearch is performed against the raw data with the query in its originalformat. The second search is performed against the rollup data and usesprocedures disclosed herein to determine which rollup job(s) to search.

The search service endpoint 112 employs logic that allows fordetermination of which of a plurality of rollup indexes are appropriatebased on the query parts. By way of example, if there are ten rollupjobs stored in a single index, which cover the raw data with varyingdegrees of completeness and different intervals, the search serviceendpoint 112 needs to determine which set of jobs to actually search.Incorrect decisions can lead to inaccurate aggregation results (e.g.over-counting doc counts, or bad metrics). Additional details regardingthe selection of rollup indices are found infra with respect to FIG. 3.

The search service endpoint 112 accepts one or more rollup indices.These can be a mix of regular, non-rollup indices and rollup indices.However, in some instances only one rollup index is specified. The exactlist of rules for the index parameter are can include but are notlimited to: (1) at least one index/index-pattern is specified. This canbe either a rollup or non-rollup index. Omitting the index parameter, orusing _all, is not permitted; (2) multiple non-rollup indices may bespecified; and (3) only one rollup index may be specified.

As noted above, the rollup service 104 can utilize different intervalgranularity than that used for collecting raw data. Rollups are storedat a certain granularity, as defined by the date_histogram group in theconfiguration. This means a user can only search/aggregate the rollupdata with an interval that is greater-than or equal to the configuredrollup interval. For example, if data is rolled up at hourly intervals,the rollup service 104 can aggregate on any time interval hourly orgreater. Intervals that are less than an hour will generate an error,since the data simply does not exist for finer granularities.

As an aside, the interval used with respect to a query is referred to insome embodiments as a third time interval. Thus, the first time intervalis with respect to how the raw data is gathered, the second timeinterval relates to the rollup job, and the third time interval relatesto the query executed against the rollup job.

The interval specified in an aggregation request can be a whole multipleof the configured interval. If the job was configured to rollup on 3 d(three day) intervals, the user can only query and aggregate onmultiples of three (3 d, 6 d, 9 d, etc.). In some embodiments it isrecommended to configure a single job with the smallest granularity thatis needed, and allow the search endpoint to function as selected at alater point in time. In some embodiments, if multiple jobs are presentin a single rollup index with varying intervals, the search serviceendpoint 112 will identify and use the job(s) with the largest intervalto satisfy the search request.

In certain embodiments, the rollup service 104 is configured to allowqueries for a term, terms, a range, and match all, and/or any compoundquery (Boolean, Boosting, ConstantScore™, and so forth). Furthermore,these queries can only use fields that were also saved in the rollup jobas a group. If you wish to filter on a keyword hostname field, thatfield must have been configured in the rollup job under a termsgrouping.

FIG. 2 illustrates an example diagrammatic example of how raw data isconverted into a rollup index using aggregation trees. In general, eachtree represents one rollup interval worth of time. In this example, twoaggregation trees 202 and 204 are illustrated, which illustrate achronological progression. The aggregation tree 202 is scheduled forcreation at a point in time that is two hours in the future, whereas theaggregation tree 204 is created one hour in the future. So when therollup service 104 is running it would obtain these two aggregationtrees 202 and 204 then work recursively iterate through the “leaf” nodesand create a rollup document for each.

Each “leaf” of the aggregation tree is a summary document of the rawdata of the branch of the tree to which it belongs. In general, the rawdata includes data elements that comprise at least one field and atleast one metric related to the at least one field.

For example, the document 206 includes information regarding time, zone,and server which produced the raw data document 206. In this instance,the data leaves/documents of Zone A generated by Servers 1 and 2 arerolled up into a rollup index 210. In this example, another document 208is generated for Server 2 of Sone A. Each of the documents 206/208represents a unique rollup job.

Each of the example documents 206 and 208 comprises the data elements ofa rollup job. The aspect of flattening of data elements involves thecreation of the aggregation trees and the grouping of data elements thatare alike. The “flattening” of the raw data occurs through the creationof the leaf documents at the bottom-most level of the aggregation tree202. The documents 206/208 can be combined into a single rollup index210 for storage and future querying.

FIG. 3 is a flow diagram of an example rollup index searching process.The method includes an initial step 302 of locating all rollup jobs in aparticular rollup index. Again, a rollup index can comprise a pluralityof rollup jobs that can be related or even orthogonal in content to oneanother.

Next, the method includes a step 304 of recursively walking the querycomponents of a query. This includes steps related to obtaining a queryand parsing the query into distinct parts or fields. Each of these partsis recursively walked with respect to the rollup jobs located in step302. At each recursive step, the method determines if the query fieldexists in the rollup index. If not the process terminates and a warningcan be presented to the user. Next, the method includes a step 306 ofrewriting the field of the query using an internal naming conventionrelated to the field names used to generate the rollup jobs. This may bean optional step in some embodiments.

Next, the method includes a step 308 of recursively walking theaggregation components which includes aggregations of data elements thatcorrespond to one another. As noted above, this could include sensordata or IP addresses. In some embodiments, the method includes a step310 of iterating through the list of rollup jobs located in step 302 andremoving any of the rollup jobs that do not satisfy the aggregationcomponent's requirements (e.g., same field type, same settings, and soforth). After iterating through all the jobs, if there are no candidatejobs left the process can terminate and a warning message can bepresented to the user.

For context, some aggregations (like a terms aggregation, or adate_histogram) can contain “children” aggregations. This could beanother terms agg, for example or it could be a “leaf” aggregation thatcontains no children (like a min, max, average, etc.) metric agg. Oncethe bottom of a tree is reached and an aggregation is found thatcontains no more children, it will be determined if there is a rollupjob that supports the data encountered up to this point in the iterativeprocess.

If so, the method includes a step 312 of returning a list of potentialmatching rollup jobs. If no leaf nodes are found, the list of candidaterollup jobs are passed to the child aggregation step above and theprocess is initiated over again. It can be assumed that all candidaterollup jobs that pass step 312 satisfy all the aggregation componentsrequirements, so they are all usable. Next, the method includes a step314 of sorting the candidate rollup jobs to identify the “best” job(s).The final, “best” candidate job is the one that will be searched andused to generate a query response.

With respect to sorting, in some embodiments, sorting criteria areutilized to sort rollup jobs. In one embodiment, rollup jobs can besorted by date_histogram interval. It will be understood that largerintervals are “better” as they provide larger aggregation pools. Ifthere are ties, the method will prefer a rollup job with a largeraverage histogram interval. Also, if there are ties, the method willprefer the rollup job with fewer terms groups.

FIG. 4 is another flowchart of an example method of the presentdisclosure. The method includes a step 402 of obtaining raw data fromone or more systems providing one or more services. As noted above theraw data comprises data elements having corresponding fields andmetrics. In general, the data elements are obtained according to a firsttime interval such as every second or every ten seconds. This creates avolume of raw data having a storage size that in some instances is quitelarge in volume.

Next, the method includes a step 404 of generating a rollup index of theraw data based on a first set of rollup parameters that compriseselections of the fields and metrics. That is, the user can select whichfields and metrics are important to them. These selections are then usedto guide the creation of rollup jobs that can be coalesced into a rollupindex. The method includes a step 406 of searching for matching ones ofthe data elements of the raw data that correspond to the first set ofrollup parameters. Next, the method includes a step 408 of grouping thedata elements together based on the matching, as well as a step 410 offlattening the data elements using a tree structure. In one or moreembodiments, the method includes a step 412 of rolling up the dataelements into the rollup index using the tree structure based on asecond time interval that is larger than the first time interval. Therollup index can be stored alongside the raw data with both beingavailable to search through a querying service.

FIG. 5 is a diagrammatic representation of an example machine in theform of a computer system 1, within which a set of instructions forcausing the machine to perform any one or more of the methodologiesdiscussed herein may be executed. In various example embodiments, themachine operates as a standalone device or may be connected (e.g.,networked) to other machines. In a networked deployment, the machine mayoperate in the capacity of a server or a client machine in aserver-client network environment, or as a peer machine in apeer-to-peer (or distributed) network environment. The machine may be,for example, a base station, a personal computer (PC), a tablet PC, aset-top box (STB), a personal digital assistant (PDA), a cellulartelephone, a portable music player (e.g., a portable hard drive audiodevice such as an Moving Picture Experts Group Audio Layer 3 (MP3)player), a web appliance, a network router, switch or bridge, or anymachine capable of executing a set of instructions (sequential orotherwise) that specify actions to be taken by that machine. Further,while only a single machine is illustrated, the term “machine” shallalso be taken to include any collection of machines that individually orjointly execute a set (or multiple sets) of instructions to perform anyone or more of the methodologies discussed herein.

The example computer system 1 includes a processor or multipleprocessors 5 (e.g., a central processing unit (CPU), a graphicsprocessing unit (GPU), or both), and a main memory 10 and static memory15, which communicate with each other via a bus 20. The computer system1 may further include a video display 35 (e.g., a liquid crystal display(LCD)). The computer system 1 may also include an alpha-numeric inputdevice(s) 30 (e.g., a keyboard), a cursor control device (e.g., amouse), a voice recognition or biometric verification unit (not shown),a drive unit 37 (also referred to as disk drive unit), a signalgeneration device 40 (e.g., a speaker), and a network interface device45. The computer system 1 may further include a data encryption module(not shown) to encrypt data.

The drive unit 37 includes a computer or machine-readable medium 50 onwhich is stored one or more sets of instructions and data structures(e.g., instructions 55) embodying or utilizing any one or more of themethodologies or functions described herein. The instructions 55 mayalso reside, completely or at least partially, within the main memory 10and/or within static memory 15 and/or within the processors 5 duringexecution thereof by the computer system 1. The main memory 10, staticmemory 15, and the processors 5 may also constitute machine-readablemedia.

The instructions 55 may further be transmitted or received over anetwork via the network interface device 45 utilizing any one of anumber of well-known transfer protocols (e.g., Hyper Text TransferProtocol (HTTP)). While the machine-readable medium 50 is shown in anexample embodiment to be a single medium, the term “computer-readablemedium” should be taken to include a single medium or multiple media(e.g., a centralized or distributed database and/or associated cachesand servers) that store the one or more sets of instructions. The term“computer-readable medium” shall also be taken to include any mediumthat is capable of storing, encoding, or carrying a set of instructionsfor execution by the machine and that causes the machine to perform anyone or more of the methodologies of the present application, or that iscapable of storing, encoding, or carrying data structures utilized by orassociated with such a set of instructions. The term “computer-readablemedium” shall accordingly be taken to include, but not be limited to,solid-state memories, optical and magnetic media, and carrier wavesignals. Such media may also include, without limitation, hard disks,floppy disks, flash memory cards, digital video disks, random accessmemory (RAM), read only memory (ROM), and the like. The exampleembodiments described herein may be implemented in an operatingenvironment comprising software installed on a computer, in hardware, orin a combination of software and hardware.

Not all components of the computer system 1 are required and thusportions of the computer system 1 can be removed if not needed, such asInput/Output (I/O) devices (e.g., input device(s) 30). One skilled inthe art will recognize that the Internet service may be configured toprovide Internet access to one or more computing devices that arecoupled to the Internet service, and that the computing devices mayinclude one or more processors, buses, memory devices, display devices,input/output devices, and the like. Furthermore, those skilled in theart may appreciate that the Internet service may be coupled to one ormore databases, repositories, servers, and the like, which may beutilized in order to implement any of the embodiments of the disclosureas described herein.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present technology has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the present technology in the form disclosed. Manymodifications and variations will be apparent to those of ordinary skillin the art without departing from the scope and spirit of the presenttechnology. Exemplary embodiments were chosen and described in order tobest explain the principles of the present technology and its practicalapplication, and to enable others of ordinary skill in the art tounderstand the present technology for various embodiments with variousmodifications as are suited to the particular use contemplated.

Aspects of the present technology are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thepresent technology. It will be understood that each block of theflowchart illustrations and/or block diagrams, and combinations ofblocks in the flowchart illustrations and/or block diagrams, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the flowchart and/or block diagram block orblocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The flowchart and block diagrams in the Figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods and computer program products according to variousembodiments of the present technology. In this regard, each block in theflowchart or block diagrams may represent a module, section, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

In the following description, for purposes of explanation and notlimitation, specific details are set forth, such as particularembodiments, procedures, techniques, etc. in order to provide a thoroughunderstanding of the present invention. However, it will be apparent toone skilled in the art that the present invention may be practiced inother embodiments that depart from these specific details.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” or“according to one embodiment” (or other phrases having similar import)at various places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments. Furthermore, depending on the context ofdiscussion herein, a singular term may include its plural forms and aplural term may include its singular form. Similarly, a hyphenated term(e.g., “on-demand”) may be occasionally interchangeably used with itsnon-hyphenated version (e.g., “on demand”), a capitalized entry (e.g.,“Software”) may be interchangeably used with its non-capitalized version(e.g., “software”), a plural term may be indicated with or without anapostrophe (e.g., PE's or PEs), and an italicized term (e.g., “N+1”) maybe interchangeably used with its non-italicized version (e.g., “N+1”).Such occasional interchangeable uses shall not be consideredinconsistent with each other.

Also, some embodiments may be described in terms of “means for”performing a task or set of tasks. It will be understood that a “meansfor” may be expressed herein in terms of a structure, such as aprocessor, a memory, an I/O device such as a camera, or combinationsthereof. Alternatively, the “means for” may include an algorithm that isdescriptive of a function or method step, while in yet other embodimentsthe “means for” is expressed in terms of a mathematical formula, prose,or as a flow chart or signal diagram.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

It is noted that the terms “coupled,” “connected”, “connecting,”“electrically connected,” etc., are used interchangeably herein togenerally refer to the condition of being electrically/electronicallyconnected. Similarly, a first entity is considered to be in“communication” with a second entity (or entities) when the first entityelectrically sends and/or receives (whether through wireline or wirelessmeans) information signals (whether containing data information ornon-data/control information) to the second entity regardless of thetype (analog or digital) of those signals. It is further noted thatvarious figures (including component diagrams) shown and discussedherein are for illustrative purpose only, and are not drawn to scale.

If any disclosures are incorporated herein by reference and suchincorporated disclosures conflict in part and/or in whole with thepresent disclosure, then to the extent of conflict, and/or broaderdisclosure, and/or broader definition of terms, the present disclosurecontrols. If such incorporated disclosures conflict in part and/or inwhole with one another, then to the extent of conflict, the later-dateddisclosure controls.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and notlimitation. The descriptions are not intended to limit the scope of theinvention to the particular forms set forth herein. To the contrary, thepresent descriptions are intended to cover such alternatives,modifications, and equivalents as may be included within the spirit andscope of the invention as defined by the appended claims and otherwiseappreciated by one of ordinary skill in the art. Thus, the breadth andscope of a preferred embodiment should not be limited by any of theabove-described exemplary embodiments.

What is claimed is:
 1. A method, comprising: obtaining raw data from oneor more systems providing one or more services, the raw data comprisingdata elements having corresponding fields and metrics, the data elementsbeing obtained according to a first time interval, the raw data having astorage size; generating a rollup index of the raw data based on a firstset of rollup parameters that comprises selections of the fields andmetrics by: searching for matching ones of the data elements of the rawdata that correspond to the first set of rollup parameters; grouping thedata elements together based on the matching; flattening the dataelements using a tree structure; and rolling up the data elements intothe rollup index using the tree structure based on a second timeinterval that is larger than the first time interval; storing the rollupindex in a storage space, the rollup index being reduced in sizerelative to the raw data so as to reduce an amount of the storage spacerequired to store rollup index relative to storing the raw data; andstoring the raw data along with the rollup index, wherein the raw datacomprises at least a last segment of the raw data covering a latterportion of the first time interval, wherein the rollup index comprisesadditional flattened data elements selected using a second set of rollupparameters, wherein the second set of rollup parameters are orthogonalin content to the first set of rollup parameters.
 2. The methodaccording to claim 1, wherein when the first time interval is measuredin milli-seconds or seconds, the second time interval is measured in anyof days, weeks, months, years, or other time frames that are larger thanseconds.
 3. The method according to claim 1, wherein the rollup index isstored alongside the raw data with both being available to searchthrough a querying service.
 4. The method according to claim 1, whereinthe selections of the fields and metrics guide a creation of at leastone rollup job that is coalesced into a rollup index.
 5. The methodaccording to claim 1, wherein the flattening of the data elementsfurther comprises a creation of the aggregation trees and a grouping ofdata elements that are alike.
 6. The method according to claim 4,wherein the at least one rollup job runs continuously in the background.7. The method according to claim 6, further comprising: receiving aquery having query parameters; parsing the query parameters; searchingthe query parameters against the rollup index; and generating a responseusing the rollup index.
 8. The method according to claim 7, furthercomprising initially searching the last segment of the raw data forportions of the data elements that correspond to the query parametersprior to searching the rollup index.
 9. The method according to claim 7,wherein the response is generated relative to a third time interval thatis larger than the second time interval.
 10. The method according toclaim 1, wherein the rollup index is continually updated as the one ormore systems generate new data elements.
 11. A system, comprising: arollup service configured to: obtain raw data from one or more systemsproviding one or more services, the raw data comprising data elementshaving corresponding fields and metrics, the data elements beingobtained according to a first time interval, the raw data having astorage size; generate a rollup index of the raw data based on a firstset of rollup parameters that comprises selections of the fields andmetrics by: searching for matching ones of the data elements of the rawdata that correspond to the first set of rollup parameters; grouping thedata elements together based on the matching; flattening the dataelements; and rolling up the data elements for placement into the rollupindex based on a second time interval that is larger than the first timeinterval; store the rollup index in a storage space, the rollup indexbeing reduced in size relative to the raw data so as to reduce an amountof the storage space required to store rollup index relative to storingthe raw data; store the raw data along with the rollup index, whereinthe raw data comprises at least a last segment of the raw data coveringa latter portion of the first time interval, wherein the rollup indexcomprises additional flattened data elements selected using a second setof rollup parameters, wherein the second set of rollup parameters areorthogonal in content to the first set of rollup parameters; and asearch service endpoint configured to: receive a query having queryparameters; parse the query parameters; search the query parametersagainst the rollup index; and generate a response using the rollupindex.
 12. The system according to claim 11, wherein the search serviceendpoint configured to is further configured to initially search a lastsegment of the raw data for portions of the data elements thatcorrespond to the query parameters prior to searching the rollup index.13. The system according to claim 12, wherein the response is generatedrelative to a third time interval that is larger than the second timeinterval.
 14. The system according to claim 13, wherein the rollup indexis continually updated as the one or more systems generate new dataelements.
 15. The system according to claim 14, wherein when the firsttime interval is measured in seconds the second time interval ismeasured in any of days, weeks, months, years, or other time frames thatare larger than seconds.
 16. The system according to claim 15, whereinthe rollup service is configured to store the raw data along with therollup index.
 17. A method, comprising: obtaining raw historical datafor a computing system, the raw historical data being obtained accordingto a first time interval; converting the raw historical data into arollup index, the rollup index comprising aggregations of data elementsof the raw historical data which are grouped according to at least onefield and at least one metric regarding the at least one field; storingthe raw historical data along with the rollup index, wherein the rawhistorical data comprises at least a last segment of the raw historicaldata covering a latter portion of the first time interval, wherein therollup index comprises additional flattened data elements selected usinga second set of rollup parameters, wherein the second set of rollupparameters are orthogonal in content to a first set of rollupparameters; and generating a query response for a query using the rollupindex, the query response substantially corresponding to a queryresponse when executed against the raw historical data due to theconversion of the raw historical data into the rollup index.
 18. Themethod according to claim 17, wherein the raw historical data isconverted into the rollup index using flattened aggregation trees. 19.The method according to claim 17, wherein the aggregations of dataelements in the rollup index are obtained relative to a second timeinterval that is greater than the first time interval, wherein the queryresponse is generated relative to a third time interval that is greaterthan or equal to the second time interval.
 20. The method according toclaim 17, wherein the rollup index is continually updated as one or moresystems generate new data elements.